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The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in a patent granted on an application for patent by another filed in the 
United States before the invention thereof by the applicant for patent, or on an international application 
by another who has fulfilled the requirements of paragraphs (1), (2), and (4) of section 371(c) of this 
title before the invention thereof by the applicant for patent. 

The changes made to 35 U.S.C. 102(e) by the American Inventors Protection Act 
of 1999 (AIPA) and the Intellectual Property and High Technology Technical 
Amendments Act of 2002 do not apply when the reference is a U.S. patent resulting 
directly or indirectly from an international application filed before November 29, 2000. 
Therefore, the prior art date of the reference is determined under 35 U.S.C. 1 02(e) prior 
to the amendment by the AIPA (pre-AlPA 35 U.S.C. 102(e)). 

Claims 1 , 4-10, and 17 are rejected under 35 U.S.C. 102(e) as being anticipated 
by Heidmann et al. (6,057,833). 

As per claim 1 , Heidmann teaches the claimed "method for enhancing video, 
comprising the steps of "identifying one or more positions in a first image" (Heidmann, 
column 12, lines 38-42); "converting said one or more positions to one or more locations 
in relation to a virtual surface, converting said one or more locations in relation to said 
virtual surface to one or more positions in a second image" (Heidmann, column 12, lines 
43-67); and "enhancing said second image based on said one or more positions in said 
second image" (Heidmann, column 14, lines 2-16). 
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Claim 4 adds into claim 1 "using a pointing device to select one or more pixels on 
a monitor" which Heidman teaches in column 4, lines 47-48. 

Claim 5 adds into claim 1 "using a pointing device to select one or more pixels on 
a monitor and using said selected pixels to determine said one or more positions in a 
first image" which Heidmann teaches in column 4, lines 61-64, column 6, lines 23-28. 

Claim 6 adds into claim 1 "selecting a region of said first image" which Heidmann 
teaches in column 6, lines 19-22. 

Claim 7 adds into claim 1 "said one or more locations are three dimensional 
locations on said virtual surface" which Heidmann teaches in column 12, lines 49-50. 

Claim 8 adds into claim 1 "said one or more locations are all on said virtual 
surface" which Heidmann teaches in column 12, lines 49-50. 

Claim 9 adds into claim 1 "said second image depicts an environment; said 
second image is captured by a camera; and said step of converting said one or more 
locations is performed without surveying a position of said camera in relation to said 
environment" which Heidmann teaches in figures 10A-10B. 

Claim 10 adds into claim 1 "said second image depicts an environment having an 
actual surface; said one or more positions in said second image are on an image of said 
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actual surface in said second image; and said step of converting said one or more 
locations are performed without knowing distance between said first actual surface and 
a camera capturing said second image" which Heidmann teaches in col. 4, lines 4-8. 

Claim 17 adds into claim 1 "the steps of converting said one or more locations in 
relation to said virtual surface to one or more positions in a third image; and enhancing 
said third image based on said one or more positions in said third image" which 
Heidmann teaches in column 9, lines 31-51. 

Applicant's arguments filed December 22, 2003 have been fully considered but 
they are not deemed to be persuasive. Applicant argues that Heidmann does not teach 
"conversion of positions identified in a first image to locations in relation to a virtual 
surface" which is not correct. Heidmann teaches "graphical elements may be drawn 
into a virtual surface which can be moved or animated in 2D or 3D, that is either scaled 
or slid around on the screen, or tumbled, viewed in perspective, warped, etc." (column 
13, lines 7-10) and the positions of the graphical elements are conversed into the virtual 
surface, creating the appearance of the graphical elements stick to the real scene. 
Applicant also goes further to discuss the properties of the virtual surfaces whose 
features do not appear on the claims. Accordingly, the claimed invention as 
represented in the claims does not represent a patentable distinction over the art of 
record. 
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The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 

the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 

the various claims was commonly owned at the time any inventions covered therein 

were made absent any evidence to the contrary. Applicant is advised of the obligation 

under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 

not commonly owned at the time a later invention was made in order for the examiner to 

consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 

prior art under 35 U.S.C. 1 03(a). 

Claims 2, 3, 11-16, 18-21,22-58 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over HEIDMANN et al. (6,057,833). 

Claim 2 adds into claim 1"said virtual surface is a sphere" which would have 
been obvious in view of Heidmann's projection surface (col. 12, lines 49-50) because 
such surface can be any type of surface such as a sphere. 

Claim 3 adds into claim 2 "said first image and said second image are video 
images; said first image depicts an environment having an actual surface (Heidmann, 
e.g., play field); said second image depicts said actual surface (Heidmann, e.g., camera 
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field); said one or more positions in said first image are on an image of said actual 
surface in said first image; said one or more locations are three dimensional locations 
on said sphere; said one or more positions in said second image are on an image of 
said actual surface in said second image (Heidmann, figures 10A-10B); said step of 
enhancing includes editing said second image in relation to said one or more positions 
in said second image, said step of converting said one or more locations is based on 
camera sensor data; and said camera sensor data pertains to an attitude of a camera 
capturing said second image" which would have been obvious to Heidmann's camera 
shot on a play field (column 12, line 35 to column 13, line 33) because the 3D surface 
can be any type of 3D surface such as a sphere. 

Claim 1 1 adds into claim 1 "said step of converting said one or more locations is 
based on camera sensor data; and said camera sensor data pertains to an attitude of a 
camera capturing said second image" which would have been obvious to Heidmann's 
camera data (column 4, lines 4-8; column 12, lines 43-57) because the camera data can 
be collected through the sensors as claimed. 

Claim 12 adds into claim 1 "said step of converting said one or more locations is 
based on camera sensor data and stabilization offset data; said camera sensor data 
pertains to an attitude of a camera capturing said second image; and said stabilization 
offset data is used to correct said camera sensor data" which would have been obvious 
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to Heidmann's camera on the play field because the accuracy of data is necessary for a 
comprehensive display of the scene (Heidmann, column 4, lines 4-15 or figs 10A-10B). 

Claim 13 adds into claim 12 "the steps of identifying positions of edges; 
transforming said positions of said edges to edge locations on said virtual surface; 
transforming said edge locations to edge positions in said second image using said 
camera sensor data; determining said stabilization -offset data based on a difference 
between said transformed edge positions in said second image and actual edge 
positions in said second image" which would have been obvious to Heidmann's camera 
on the play field because the accuracy of data is necessary for a comprehensive display 
of the scene (Heidmann, column 4, lines 4-15, column 6, line 53 to column 7, line 27). 

Claim 14 adds into claim 1 "said step of enhancing includes adding an 
enhancement image to said second image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
transparency scene (col. 6, line 64 to column 7, line 1 ) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 



Claim 15 adds into claim 14 "said step of accounting for occlusions of said 
enhancement image includes accessing a color map and editing pixels of said second 
image according to said color map" which Heidmann teaches in col. 5, lines 63-65 in 



Application/Control Number: 09/928,165 Page 8 

Art Unit: 2671 

which the attribute data of the inserted object can be any well known type of 3D object 
attribute such as color, texture, depth, ... 

Claim 16 adds into claim 14 that "said second image is an image of an 
environment having an actual surface; and said enhancement image appears to be on 
said actual surface" which Heidmann teaches in figures 10A-10B. 

Claim 18 adds into claim 1 "said first image and said second image are video 
images; said first image depicts an environment having an actual surface; said second 
image depicts said actual surface; said step of identifying includes annotating an image 
of said actual surface in said first image, said one or more positions represent said 
annotation, said annotation is at a first orientation in relation to said actual surface; and 
said step of enhancing includes adding said annotation to an image of said actual 
surface in said second image at said first orientation in relation to said actual surface" 
which would have been obvious to Heidmann's inserted objects and animation (column 
3 line 56 to column 4, line 14) because one of those inserted objects can be an 
annotation. 

Claim 19 adds into claim 18 "said step of annotating an image includes receiving 
a set of points from a touch screen" which Heidmann teaches in column 8, lines 44-46. 
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Claim 20 adds into claim 1 "said step of identifying one or more positions a first 
image include identifying a position to add a virtual advertisement; said step of 
converting said one or more locations includes determining where to add said virtual 
advertisement into said second image; and said step of enhancing includes adding said 
virtual advertisement into said second image" which would have been obvious to 
Heidmann's inserted objects (column 4, lines 1-14) because the inserted object can be 
"a virtual advertisement". 

Claim 21 adds into claim 1 "said step of identifying one or more positions in a first 
image include identifying a set of positions in an environment, said set of positions 
bound a virtual advertisement; said step of converting said one or more positions 
includes transforming said set of positions in said environment to locations on said 
virtual surface; said step of converting said one or more locations includes determining 
where said set of positions are in said second image; and said step of enhancing 
includes adding said virtual advertisement into said second image where said set of 
positions are in said second image" which would have been obvious to Heidmann's 
inserted objects (column 4, lines 1-14) because the inserted object can be "a virtual 
advertisement" which can be added to the transformed scene. 

Claim 45 adds into claim 1 "a virtual advertisement" which would have been 
obvious because Heidmann's animated object are any type of objects inserted into the 
actual surface including an advertisement to enhance the screen of the video image. 
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Claim 46 adds into claim 1 "first image depicts an environment, and the virtual 
surface is of a different shape than said actual surface" which would have been obvious 
because Heidmann's projection of the locations of an actual surface into the virtual 
surface which can be any 3D surface (e.g., floating a distance above the actual surface, 
Heidmann, column 12, lines 43-50). 

Claim 47 adds into claim 1 the positions in the first and second images are 2D 
positions and the locations are 3D locations in relation to the virtual surface" which 
Heidmann teaches in the projected 2D video images and the virtual 3D surface 
(Heidmann, column 12, lines 43-50). 

Claim 48 adds into claim 1 "at least one of said locations are not located on said 
virtual surface" which would have been obvious because the 3D animated object can be 
projected into the virtual surface but not actually on the surface. 

As per claim 22, Heidmann teaches the claimed "method for enhancing video" 
comprising the steps of "determining one or more locations in relation to a virtual 
surface" (Heidmann, column 12, lines 35-59); and "converting said one or more 
locations in relation to said virtual surface to one or more positions in a video image" 
(Heidmann, column 12, line 60 to column 13, line 25). Applicant argues that Heidmann 
does not teach "conversion of positions identified in a first image to locations in relation 
to a virtual surface" which is not correct. Heidmann teaches "graphical elements may 
be drawn into a virtual surface which can be moved or animated in 2D or 3D, that is 
either scaled or slid around on the screen, or tumbled, viewed in perspective, warped, 
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etc." (column 13, lines 7-10) and the positions of the graphical elements are conversed 
into the virtual surface, creating the appearance of the graphical elements stick to the 
real scene. Applicant amends the claim and argues that Heidmann does not teach 
"said virtual surface is different from said actual surface depicted in said first video 
image". However, such claimed feature would have been obvious in view of 
Heidmann's projection of the locations of an actual surface into the virtual surface 
because Heidmann virtual surface can be any 3D surface (e.g., floating a distance 
above the actual surface, Heidmann, column 12, lines 43-50). 

Claim 23 adds into claim 22 "the steps enhancing said video image based on 
said one or more positions in said video image" which Heidmann teaches in column 8, 
lines 44-46. 



Claim 24 adds into claim 23 "said step of enhancing includes adding an 
enhancement image to said video image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
transparency scene (col. 6, line 64 to column 7, line 1) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 

Claim 25 adds into claim 22 "said virtual surface is a sphere; and said one or 
more locations are three dimensional locations on said sphere" which would have been 
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obvious to Heidmann's camera shot on a play field (column 12, line 35 to column 13, 
line 33) because the 3D surface can be any type of 3D surface such as a sphere. 

Claim 26 adds into claim 22 "said video image depicts an environment; said 
video image is captured by a first camera; and said step of converting said one or more 
locations is performed without knowing a position of said first camera in relation to said 
environment" which would have been obvious to Heidmann's camera on the play field 
because the accuracy of data is necessary for a comprehensive display of the scene 
(Heidmann, column 4, lines 4-15, column 6, line 53 to column 7, line 27). 

Claim 49 adds into claim 22 "3D locations and 2D positions" which Heidmann 
teaches in the projected 2D video images and the virtual 3D surface (Heidmann, column 
12, lines 43-50). 

Claim 50 adds into claim 22 "3D actual and virtual surfaces and 2D video 
images" which Heidmann teaches in the 3D environment and 3D virtual surface (column 
12, lines 43-51) and 2D video signals of the actual surface. 

Claim 51 is the same as claim 1 and adds "defining a virtual surface, wherein the 
portion of said virtual surface is defined at a desired position in proximity to a portion of 
an environment to be enhanced" which Heidmann teaches in column 12, lines 45-48. 

Claim 52 adds into claim 51 "an enhancement image" which Heidmann teaches 
in the animated object 9column 12, lines 43-59. 
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Claim 53 adds into claim 52 "said enhancement image appears to be located at 
said portion of said environment" which Heidmann teaches in figures 10A-10B. 

Claims 54-58 claim the features of claims 45-50 which have been rejected 
above; therefore, they are rejected under the same reason. 

As per claim 27, Heidmann teaches the claimed "one or more processor 
readable storage devices" for storing processor readable code, said processor readable 
code for programming one or more processors to perform a method comprising the 
steps of "determining one or more locations in relation to a virtual surface" (Heidmann, 
column 12, lines 35-59); and "converting said one or more locations in relation to said 
virtual surface to one or more positions in a video image" (Heidmann, column 12, line 60 
to column 13, line 25). Applicant argues that Heidmann does not teach "conversion of 
positions identified in a first image to locations in relation to a virtual surface" which is 
not correct. Heidmann teaches "graphical elements may be drawn into a virtual surface 
which can be moved or animated in 2D or 3D, that is either scaled or slid around on the 
screen, or tumbled, viewed in perspective, warped, etc." (column 13, lines 7-10) and the 
positions of the graphical elements are conversed into the virtual surface, creating the 
appearance of the graphical elements stick to the real scene. Applicant amends the 
claim and argues that Heidmann does not teach "said virtual surface is of a different 
shape than said actual surface depicted in said first video image". However, such 
claimed feature would have been obvious in view of Heidmann's projection of the 
locations of an actual surface into the virtual surface because Heidmann virtual surface 
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can be any 3D surface (e.g., floating a distance above the actual surface, Heidmann, 
column 12, lines 43-50). 

Claim 28 adds into claim 27 "the steps of: enhancing said video image based on 
said one or more positions in said video image" which Heidmann teaches in column 8, 
lines 44-46. 

Claim 29 adds into claim 28 "said step of enhancing includes adding an 
enhancement image to said video image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
transparency scene (col. 6, line 64 to column 7, line 1) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 

Claim 30 adds into claim 27 "said virtual surface is a sphere; and said one or 
more locations are three dimensional locations on said sphere" which would have been 
obvious to Heidmann's camera shot on a play field (column 12, line 35 to column 13, 
line 33) because the 3D surface can be any type of 3D surface such as a sphere. 



Claim 31 adds into claim 27 "said video image depicts an environment; said 
video image is captured by a first camera; and said step of converting said one or more 
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locations is performed without knowing a position of said first camera in relation to said 
environment" which would have been obvious to Heidmann's camera on the play field 
because the accuracy of data is necessary for a comprehensive display of the scene 
(Heidmann, column 4, lines 4-15, column 6, line 53 to column 7, line 27). 

Claim 32 adds into claim 27 "the steps of storing one or more positions in a first 
image, said step of determining includes converting said one or more positions to said 
one or more locations in relation to said virtual surface; and enhancing said video image 
based on said one or more positions in said video image" which Heidmann teaches in 
column 4, lines 61-64, column 6, lines 23-28. 

Claim 33 adds into claim 32 "said first image is a video image; said first image 
depicts an environment having an actual surface; said video image depicts said actual 
surface; said step of storing includes storing an annotation of an image of said actual 
surface, said one or more positions represent said annotation, said annotation is at a 
first orientation in relation to said actual surface; and said step of enhancing includes 
adding said annotation to an image of said actual surface in said video image at said 
first orientation in relation to said actual surface" which would have been obvious to 
Heidmann's inserted objects and animation (column 3 line 56 to column 4, line 14) 
because one of those inserted objects can be an annotation. 
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Claim 34 adds into claim 32 "said step of storing includes storing a set of 
positions in an environment, said set of position bound a virtual advertisement; said step 
of converting said one or more positions includes transforming said set of positions in 
said environment to locations on said virtual surface; said step of converting said one or 
more locations includes determining where said set of positions are in said second 
image; and said step of enhancing includes adding said virtual advertisement into said 
second image where said set of positions are in said second image" which would have 
been obvious to Heidmann's inserted objects (column 4, lines 1-14) because the 
inserted object can be "a virtual advertisement". 

As per claim 35, Heidmann teaches the claimed "apparatus for enhancing video" 
comprising "video modification hardware" (Heidmann, column 12, lines 35-59); and one 
or more processing devices in communication with said video modification hardware, 
said one or more processing devices perform a method comprising the steps of 
determining one or more locations in relation to a virtual surface, and converting said 
one or more locations in relation to said virtual surface to one or more positions in a 
video image (Heidmann, column 12, line 60 to column 13, line 25). 

Claim 36 adds into claim 35 "the steps of enhancing said video image based on 
said one or more positions in said video image" which Heidmann teaches in column 8, 
lines 44-46. 
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Claim 37 adds into claim 35 "said step of enhancing includes adding an 
enhancement image to said video image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
transparency scene (col. 6, line 64 to column 7, line 1 ) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 

Claim 38 adds into claim 35 "said virtual surface is a sphere; and said one or 
more locations are three dimensional locations on said sphere" which would have been 
obvious to Heidmann's camera shot on a play field (column 12, line 35 to column 13, 
line 33) because the 3D surface can be any type of 3D surface such as a sphere. 

Claim 39 adds into claim 35 "said video image depicts an environment; said 
video image is captured by a first camera; and said step of converting said one or more 
locations is performed without knowing a position of said first camera in relation to said 
environment" which would have been obvious to Heidmann's camera on the play field 
because the accuracy of data is necessary for a comprehensive display of the scene 
(Heidmann, column 4, lines 4-15, column 6, line 53 to column 7, line 27). 

Claim 40 adds into claim 35 "the steps of storing one or more positions in a first 
image, said step of determining includes converting said one or more positions to said 
one or more locations in relation to said virtual surface; and enhancing said video image 
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based on said one or more positions in said video image" which Heidmann teaches in 
column 4, lines 61-64, column 6, lines 23-28. 

Claim 41 adds into claim 40 "said first image is a video image; said first image 
depicts an environment having an actual surface; said video image depicts said actual 
surface; said step of storing includes storing an annotation on an image of said actual 
surface in said first image, said one or more positions represent said annotation, said 
annotation is at a first orientation in relation to said actual surface; and said step of 
enhancing includes adding said annotation to an image of said actual surface in said 
video image at said first orientation in relation to said actual surface" which would have 
been obvious to Heidmann's inserted objects and animation (column 3 line 56 to column 
4, line 14) because one of those inserted objects can be an annotation. 

Claim 42 adds into claim 40 "storing a set of positions in an environment, said set 
of position bound a virtual advertisement, said step of converting said one or more 
positions includes transforming said set of positions in said environment to locations on 
said virtual surface; said step of converting said one or more locations includes 
determining where said set of positions are in said video image; and said step of 
enhancing includes adding said virtual advertisement into said video image where said 
set of positions are in said video image" which would have been obvious to Heidmann's 
inserted objects (column 4, lines 1-14) because the inserted object can be "a virtual 
advertisement". 
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Claim 43 adds into claim 35 "camera sensors; a gatherer computer, said gatherer 
computer receives camera sensor data from said camera sensors; a time code inserter, 
said time code inserter receives video and adds time codes to said video; and a touch 
screen, said one or more processing devices include a first processor and a second 
processor, said first processor receives camera sensor data from said gatherer and 
video from said time code inserter, said first processor also receives annotation data 
from said touch screen, said annotation data relates to a set of positions on said touch 
screen, said first processor converts said positions on said touch screen to said one or 
more locations in relation to said virtual surface, said first processor sends drawing 
information to said second processor, said drawing information is based on said one or 
more positions in said video image, said second processor directs said video 
modification hardware to add one or more annotation image to said video image, said 
annotation image corresponds to said annotation data, said step of converting said one 
or more locations is based on said camera sensor data" which would have been 
obvious to Heidmann's inserted objects and animation (column 3 line 56 to column 4, 
line 14) because one of those inserted objects can be an annotation. 



Claim 44 adds into claim 35 "camera sensors, said camera sensors provide 
camera sensor data to said one or more processors, said step of converting said one or 
more locations is based on said camera sensor data" which would have been obvious to 
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Heidmann's camera data (column 4, lines 4-8; column 12, lines 43-57) because the 
camera data can be collected through the sensors as claimed. 

The following is a quotation of the first paragraph of 35 U.S.C. 1 1 2: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

Claim 48 is rejected under 35 U.S.C. 112, first paragraph, as failing to comply 
with the written description requirement. The claim(s) contains subject matter which 
was not described in the specification in such a way as to reasonably convey to one 
skilled in the relevant art that the inventor(s), at the time the application was filed, had 
possession of the claimed invention. It is unclear as how the disclosure teaches the 
claimed "at least one of said one or more locations are not located on said virtual 
surface". 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
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shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Phu K. Nguyen whose telephone number is (703)305 - 
9796. The examiner can normally be reached on M-F 8:00-4:30. 

The fax phone number for the organization where this application or proceeding 
is assigned is 703-872-9306. Information regarding the status of an application may be 
obtained from the Patent Application Information Retrieval (PAIR) system. Status 
information for published applications may be obtained from either Private PAIR or 
Public PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR system, 
contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Phu K. Nguyen 
March 7, 2004 
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